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Abstract — Integrating volatile renewable energy resources 
into the bulk power grid is challenging, due to the reliability 
requirement that at each instant the load and generation in 
the system remain balanced. In this study, we tackle this 
challenge for smart grid with integrated wind generation, by 
leveraging multi-timescale dispatch and scheduling. Specifically, 
we consider smart grids with two classes of energy users - 
traditional energy users and opportunistic energy users (e.g., 
smart meters or smart appliances), and investigate pricing and 
dispatch at two timescales, via day-ahead scheduling and real- 
time scheduling. In day-ahead scheduling, with the statistical 
information on wind generation and energy demands, we char- 
acterize the optimal procurement of the energy supply and the 
day-ahead retail price for the traditional energy users; in real- 
time scheduling, with the realization of wind generation and the 
load of traditional energy users, we optimize real-time prices to 
manage the opportunistic energy users so as to achieve system- 
wide reliability. More specifically, when the opportunistic users 
are non-persistent, i.e., a subset of them leave the power market 
when the real-time price is not acceptable, we obtain closed- 
form solutions to the two-level scheduling problem. For the 
persistent case, we treat the scheduling problem as a multi- 
timescale Markov decision process. We show that it can be 
recast, explicitly, as a classic Markov decision process with 
continuous state and action spaces, the solution to which can 
be found via standard techniques. 

We conclude that the proposed multi-scale dispatch and 
scheduling with real-time pricing can effectively address the 
volatility and uncertainty of wind generation and energy de- 
mand, and has the potential to improve the penetration of 
renewable energy into smart grids. 

I. Introduction 

A. Motivation 

To address the grand challenge of a sustainable energy 
industry, there has recently been a surge of interest in 
alternative energy resources, including wind, solar, bio-fuel, 
and geothermal energy. Ultimately, all these energy solutions 
hinge heavily on smart grid technologies that are capable 
of coordinating and managing dynamically interacting power 
grid participants. There is therefore an urgent need to de- 
velop a new generation of cyber-enabled energy management 
system (EMS) and supervisory control and data acquisition 
(SCADA), which can carry out reliable and possibly dis- 
tributed management of these energy sources. 

For normal operations of power systems, the precise bal- 
ance between energy supply and demand is of the most 



significance to system reliability. Integrating a large amount 
of intermittent renewable energy resources (e.g., wind gener- 
ation) into the bulk power grid has put forth great challenges 
for generation planning and system reliability. In particular, 
a complication that arises is that some primary elements of 
renewable energy sources, such as wind and solar power, 
are highly variable (stochastic) and often uncontrollable, 
making it difficult to guarantee that the load and generation 
in the system remain balanced at each instant. A mismatch 
between supply and demand could cause a deviation of 
zonal frequency from nominal value 0"), and when it gets 
severe power outages and blackouts may occur. Further, wind 
generation is non-dispatchable, in the sense that the output 
of wind turbines must be taken by all rather than by request. 
Moreover, the volatility and intermittence makes it difficult 
for system operators to obtain accurate knowledge of future 
wind generations. Traditionally, system operators maintain 
additional generation capacity and reserves (on-line or fast- 
start), at additional costs, to address the supply uncertainty. 

While the volatility of wind generation induces uncertain- 
ties in the supply side of the power grid, an emerging class 
of energy users, namely opportunistic energy users, induce 
uncertainties on the demand side as well. It is noted in (2) 
that over 10% daily energy consumption in United States 
is from the usage of appliances such as water heater, cloth 
dryers, and dish washers, which are envisaged to become 
smart and be branded as opportunistic energy users, with the 
following behaviors distinct from traditional energy users: 
1) they access the energy system in an opportunistic manner, 
according to the availability of system resources; 2) different 
from the 'always-on' demand of traditional energy users, the 
load profiles of opportunistic energy users can be bursty and 
can be either inelastic or elastic; 3) opportunistic energy users 
respond to the power market on a much finer timescale. 

The prevalence of the new class of opportunistic energy 
users, if utilized intelligently, makes demand side manage- 
ment (DSM) a promising solution to reduce the costs incurred 
by the deep penetration of wind generation |f3] . Traditionally, 
some energy users (e.g., residential and small commercial 
users) pay a fixed price per unit of electricity that is estab- 
lished to represent an average cost of power generation over 
a given time-frame (e.g., a month), independent of the gener- 
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ation cost. In contrast, under price-based DSM programs |4|, 
the retail prices are tied with the generation cost and may 
vary according to the availability of energy supplies. Often 
times, the energy market consists of a day-ahead market and 
a real-time market for electricity. Simply put, the day-ahead 
market produces financially binding schedules for the energy 
generation and consumption one day before the operating 
day. Further, the real-time market is used to tune the balance 
between the energy amount scheduled day-ahead and the 
real-time load. However, it is known that existing dynamic 
pricing mechanisms (term of use, critical peak pricing, etc) 
do not work well for handling generation uncertainty and 
managing the demand uncertainty in a real-time manner (i.e., 
within minutes). 

In this study, we will explore multi-timescale dispatch 
and scheduling to address the following challenges: 1) the 
supply uncertainty as a result of the volatility and non- 
stationarity of wind generation; 2) the demand uncertainty 
due to a large number of opportunistic energy users and 
their stochastic behaviors; 3) the coupling between sequential 
decisions across multiple timescales. 

B. Summary of Main Contributions 

Aiming to tackle the challenge of integrating volatile wind 
generation into the bulk power grid, we study dispatch and 
scheduling, for a smart grid model with two classes of 
energy users, namely traditional energy users and opportunis- 
tic energy users (e.g., smart meters or smart appliances). 
We consider a power grid with both conventional energy 
sources (e.g., thermal) and wind generation. Notably, wind 
generation is among the renewable resources that has most 
variability and uncertainty, and exhibits multi-level dynamics 
across time. To enhance the penetration of wind energy, 
we study multi-timescale dispatch and scheduling based on 
a marriage of real-time pricing and multi-settlement power 
market economics. 

Specifically, the system controller performs scheduling at 
two timescales. In the day-ahead schedule, with the statistical 
information on wind generation and energy demands, the 
operator optimally procures conventional energy supply and 
decides the optimal retail price for the traditional energy 
users, for the next day. In the real-time schedule, upon the 
realization of the wind energy generation and the demand 
from traditional energy users (which is stochastically depen- 
dent on the day-ahead retail price), the controller decides the 
real-time retail price for the opportunistic energy users. 

In particular, we explore multi-scale scheduling for two 
types of opportunistic energy users: the non-persistent and 
the persistent users. The non-persistent users leave the power 
market when they find that the current real-time price is 
unacceptable, whereas the persistent opportunistic users wait 
for the next acceptable real-time price. We obtain closed-form 
solutions for the scheduling problem when the users are non- 
persistent. For the persistent case, the scheduling problem is 
a multi-timescale Markov decision process (MMDP) that we 



recast, explicitly, as a standard Markov decision process that 
can be solved via standard solution techniques. We demon- 
strate, via numerical experiments, that the proposed two- 
timescale dispatch and scheduling enables the penetration of 
wind generation and hence improves the overall efficiency, 
by enabling two-way energy exchange between providers and 
customers, and by facilitating both information interaction as 
well as energy interaction. 

C. Related Work 

Roughly speaking, related work falls into two major cate- 
gories: the scheduling of power systems with wind generation 
integration; and the pricing and management of opportunistic 
users. 

Very recent work J5| proposed risk-limiting dispatch in 
contrast to worst-case dispatch for traditional generation 
planning, by treating scheduling with known demand and 
uncertain supply as a multi-stage decision problem with re- 
course. With the same spirit, J6) proposed stochastic security 
criterion in scheduling with wind generation. Specifically, 
a scenario-based approach (a scenario is defined as the 
trajectory of potential realization of future supply) is used 
to minimize the expected aggregated social cost incurred by 
all the scenarios in the scheduling horizon. Model predic- 
tive dispatch of wind generation was studied in |71, which 
treats future wind generation in the scheduling horizon as 
a dynamic process specified by an ARMA model, aiming 
to minimize the total generation cost subject to the physical 
constraints of conventional generators. In a nutshell, all the 
works noted above focus on managing the energy supply and 
treat the energy demand as known (or statistically known) but 
uncontrollable. 

Real-time pricing in related works J4), f3l takes place on 
the timescale of hours in conventional power systems, and 
the price response of energy users are understood with some 
high-level models. For a cyber-enabled energy system with 
a large number of opportunistic energy users, clearly it is 
more desirable to accomplish the price response and manage 
the energy demand in a much finer timescale (e.g. minutes). 
Indeed, this is one salient feature envisioned for future smart 
grids, although this poses significant challenges for planning, 
modeling, and controlling energy generation, transmission 
and distribution. 

The remainder of the paper is organized as follows. In 
Section II, we describe the energy system model and give 
a brief overview of the two-timescale settlement power 
market. We study two-timescale dispatch and scheduling 
when the opportunistic users are non-persistent in Section III. 
In Section IV, we consider persistent opportunistic users 
and formulate the scheduling problem as a multi-timescale 
Markov decision process. We provide concluding remarks 
and identify directions for future research in Section V. 

II. System Model and Problem Formulation 

We divide a 24-hour period into M slots of length T\ each, 
where M > 1; and each T\-slot, in turn, consists of K slots, 
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Fig. 1 . An energy management system for smart grid with wind generation 
and real-time pricing 



each of length T2, with K > 1. For example, the Ti-slots can 
correspond to hours in a day and the T2 -slots can correspond 
to minutes. We begin with an introduction to the supply side 
of the power system. 

A. Energy Sources and Generation Costs 

We consider a power grid with two kinds of energy 
sources: the conventional energy (e.g., thermal) and the wind 
energy. The conventional energy is, in turn, drawn from two 
sources: base-load generation and peaking generation, with 
generation cost c\ and c% per unit, respectiveljfl. Peaking 
generation is typically from fast-start generators (e.g., gas 
turbines), with a higher generation cost (C2 > ci). Due 
to the start-up time and ramp rate of generators, the base- 
load generators are scheduled day-ahead (24-hours ahead of 
the corresponding time) for each T\ slot of the next day, 
and the generation cost c\ contains start-up cost c p and 
other operating costs. In real-time scheduling of each T 2 
slot, peaking generation and wind generation are used, as 
needed, to clear the balance between demand and the base- 
load generation. 

Let D denote the aggregate energy demand (from both 
traditional and opportunistic users) and W be the wind 
generation amount, in a T2-SI0L Let L = D — W denote the 
net demand for conventional energy in a T2 slot. Suppose 
that the system operator schedules a base-load generation 
amount of £ for a Ti-slot and thus schedules s = for 
each T2-slot within the Ti-slot, for the next day. In real-time 
scheduling, the system operator need to balance demand and 
supply: 1) If L > s, then system operator dispatches a base- 
load generation of s and a peaking generation of L — ,s; 2) 
Otherwise, the over-scheduled generation incurs a start-up 
cost of c p per unit. 

B. Day-Ahead Pricing and Real-Time Pricing 

As is standard, dynamic pricing contracts between the sys- 
tem operator and end-users are used to manage the demand of 
both traditional energy users and opportunistic energy users, 
on both T\ and T2 timescales. We consider the following 
two-timescale pricing model: 

'The wind energy is assumed to be cost-free, based on (SJ. 



• Traditional energy users and opportunistic energy users 
have separate pricing contracts: day-ahead pricing for 
traditional users and real-time pricing for opportunistic 
users, respectively; 

• Day-ahead retail price: Traditional energy users are 
informed, one day ahead, of the day-ahead prices u 
corresponding to each Ti-slot; 

• Real-time retail price: opportunistic energy users re- 
ceive the real-time retail prices v at the beginning of 
each T2-slot; 

• Both retail prices have price cap u cap and v cap , respec- 
tively. 

Note that besides the day-ahead pricing model for the tradi- 
tional users, which was proposed in Q, we take a forward- 
looking perspective to identify a new class of users, namely 
the opportunistic energy users, and devise a real-time pricing 
model for these users that is cognizant of the uncertainties in 
the demand and supply. We discuss these uncertainties next. 

C. Supply Uncertainty of Wind Generation 

Wind generation is determined by the geographical and 
meteorological conditions, and may assume high fluctuations 
or relative steady patterns at different time-scales. Therefore, 
wind generation are generally considered to be non-stationary 
and volatile. Based on recent works (9), ifTOl . the wind 
generation can be modeled as a non-stationary Gaussian 
random process across T\-slots, i.e., the wind generation 
amount in fcth T2-slot of mth T\-slot is given by 



(1) 



where 6 m is the mean and a 2 is the variance. The statistical 
information is available, one day ahead, to the day-ahead 
scheduler. Indeed, this information is commonly provided by 
the forecasting functions of EMS or commercial entities. 

D. Demand Uncertainty under Real-time Pricing 

In a T2-slot, the aggregated demand from energy users, 
i.e., D, consists of two components: D t from the traditional 
energy users, and D a from the opportunistic energy users. 

Demand model for traditional energy users: The demands 
of traditional energy users could be known with reasonable 
accuracy, and the short-term price response of demand is 
well understood ifTTI . lfl2l . Based on fT3l . we model the 
energy demand of traditional energy users D t in a T2-slot 
as a random variable with mean depending on the day-ahead 
price u: 

D t = E [D t ] + e t , (2) 



with 



E [D t ] 



a t u 



it 



(3) 



where e t accounts for the uncertainty of D t and ~f t is the price 
elasticity of the traditional energy users at the corresponding 
Ti-slot. Price elasticity is traditionally used to characterize 
the price response of energy users and is formally defined as 
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the ratio of marginal percentage change in demand to that of 
price, i.e., 

(4) 



Non-stationary wind generation 



7t D t du ' 

Note that, for power systems, the price elasticity is negative, 
and a t is the normalizing constant. Worth noting is that 
a t and j t could be different for each of the T\ -slots, 
since traditional energy users' demand and persistence for 
consumption may depend on the term of the day (noon or 
evening, peaking or off-peaking hours). 

Demand model for opportunistic energy users: Under real- 
time pricing, we assume that opportunistic energy users have 
the following behaviors: 

• Opportunistic energy users arrive according to a Poisson 
process with rate Ao, which is constant within a Ti-slot 
but can vary across the T\ -slots; 

• Opportunistic energy users choose the access time ran- 
domly and independently; 

• In each T2-slot, an opportunistic user i arriving in the 
system decides to accept or reject the announced real- 
time price v by comparing with a price acceptance level 
Vi. This price acceptance level is randomly chosen and 
is i.i.d across the opportunistic users. Thus with N 
denoting the total number of opportunistic energy users 
in a T2-slot, the number of active opportunistic users, 
N a , is given by 



N 

N a = ^2 l{Vi>«}i 



(5) 



• Each active opportunistic energy user has a per-unit 
energy consumption of E . Thus the total energy de- 
mand from opportunistic energy users is given by D Q = 

N a E ; 

• Under real-time pricing, the response of opportunistic 
energy users may vary according to the applications. 
Recent study ifTTl suggests that households respond to 
high energy prices through energy conservation with 
no load shifting, while fl2l finds that most of the 
commercial customers respond by load shifting to a 
later time. With these insights, we consider two kinds of 
opportunistic energy users: non-persistent and persistent, 
of which the former leaves the system if the real-time 
price is unacceptable, while the latter waits in the system 
for a new real-time price in the next T2-SI0L 

E. General Problem Formulation 

As illustrated in Fig. |2j we consider day-ahead scheduling 
and real-time scheduling with non-stationary wind generation 
over a horizon of M T\ -slots, with the main objective being 
to maximize the overall expected profit. We elaborate further 
on multi-timescale dispatch and scheduling below. 

In day-ahead scheduling, with the distributional informa- 
tion of next-day wind generation, the system operator aims to 
find a policy n that dictates the two-timescale decisions S, u, 
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Fig. 2. Multiple timescale scheduling with non-stationary wind generation 
and real-time pricing 



and v. A general formulation of the two-timescale scheduling 
problem is provided below: 



M 



V: max]TiC(7r), 



(6) 



where, R^tc) is the total profit in a T\-slot under policy 7r, 
given by 



K 



(7) 



k=l 



where R l k is the net profit in the A;th T2-slot of the mth 
Ti-slot (henceforth called the (k, m)th slot) and ip l k is the 
system state in the (fc, m)th slot, which is observable in 
real time. When the opportunistic users are non-persistent, 
ip k m consists of the wind energy and the energy demand 
from traditional users. When the opportunistic users are 
persistent, i/j l k m consists of wind generation and traditional 
users' energy demand, as well as the energy requests of 
opportunistic users carried over from the previous T2-SI0L 

III. Dispatch and Scheduling with Non-persistent 
Opportunistic Energy Users 

Note that the energy procurement and retail price in day- 
ahead schedule have significant impact on the real-time 
retail price. The real-time pricing policy, in turn, affects the 
optimization of the day-ahead schedule. This tight coupling, 
underscores the need for joint optimization of the day-ahead 
and real-time schedules. To this end, we take a "bottom-up" 
approach in formulating the two-level scheduling problem. 
Specifically, we first formulate the real-time scheduling prob- 
lem, conditioned on the day-ahead scheduling decisions S 
and u\ and then introduce the day-ahead scheduling problem 
while taking into account the real-time scheduling policy. 

A. Real-time Scheduling on Timescale T2 

Given the amount of conventional energy procurement S 
and the day-ahead price u settled in day-ahead scheduling, 
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together with the realizations of wind generation W and tra- 
ditional energy demand D t , the real-time scheduling problem 
in a T2-slot is formulated as: 



RT 



max R (tp , s, u, v), 



(8) 



non-pst 

where ip l = {W,D t } is the system state and 

R l (ip l ,s,u,v) = uD t +E v Do [vD + l A (~c p s) 
+ CpC - Ci(s - e)) 
+ l c (-cis + c 2 e)} , (9) 

where, recall that D denotes the energy demand of op- 
portunistic energy users. The quantity e denotes the surplus 
energy, given by 



W 



(D t + D ). 



(10) 



The indicator function 1a corresponds to the event when the 
wind energy is sufficient to meet the demands of both types of 
users. Indicator 1b refers to the event when the wind energy 
is not sufficient but the total of the scheduled traditional 
energy and the wind energy is higher than the aggregate 
demand D t + D„, which necessitates the cancelation of part 
of the scheduled generation from the base-line generators 
incurring a penalty c p per unit for the canceled amount of 
generation. Indicator lc corresponds to the event that the to- 
tal of scheduled baseline generation and the wind generation 
is insufficient to meet the aggregate energy demand and the 
controller must purchase the deficit energy from fast start-up 
generators at a cost c 2 . Formally, the events are described 
below: 

1 if W > D t + D 
otherwise 



(11) 



1 if 1 A = and e > 
otherwise 

1 if e < 
otherwise 



(12) 



(13) 



B. Day-ahead Scheduling on Timescale T\ 

The day-ahead scheduling problem in the non-persistence 
case is formally given by 

K 



DA 

non — pst 



: max > Eiv, E 

S,u ^ 



w k &b tl , max [Rk(i>l, s i u ' v k)] - (14) 



where the inner maximization corresponds to the real-time 
scheduling studied above. Since Wk and D tk are indepen- 
dent and identically distributed across the T2-slots, the day- 
ahead scheduling problem in the non-persistent case can be 
optimized by simply considering the snapshot problem in a 
specific T2-slot, given b>@: 



V, 



DA 

non— ps 



t : maxEvyE^, max \R l {ip l , s, u, v)] . (15) 

S,u v 



C. Approximate Solutions 

The tight coupling between the day-ahead and real-time 
scheduling problems, along with the convolved nature of the 
uncertainties involved, makes a direct joint optimization chal- 
lenging. We therefore take an alternate approach and obtain 
approximate solutions to the real-time schedule. Based on the 
approximate real-time schedule, we propose solutions to the 
day-ahead schedule. In light of the characteristics of practical 
systems, we impose the following general conditions first. 

• Condition A: Wind generation is not sufficient to meet 
the total energy demand in the system. 

Needless to say, under condition A, 1a = in (|9). Define 
Y = s + W — Dt as the energy procurement for the demand 
of opportunistic energy users, then (0 reduces to 



R l W , a, u, v) = V v Do [(v - c 2 )D - cl B (Y - Do)] 



uD t - cxs + c 2 Y, 



(16) 



where c 



A 



ci + c 2 . 



We begin with an analysis of the real-time scheduling prob- 
lem. Recall that the number of non-persistent opportunistic 
energy users arriving in the T2-slot has a Poisson distribution, 
with mean Ki = \ Ti- It follows that the number of users 
that become active, denoted as N a , is also a Poisson random 
variable with mean «iP(V > v). 

Note that the price elasticity of the opportunistic energy 
users, 7 is given by 

v dN a 



lo 



(17) 



N a dv ■ 

The opportunistic energy users are said to be relatively 
inelastic if — 1 < j a < 0, i.e., the percentage change in 
demand is greater than that of price; otherwise, they are 
relatively elastic. Further, let v m i n be the highest price that 
is acceptable to all opportunistic energy users. 

Then, the solution to ( TTTb is given by: 



N a = N (v/v, 
Based on ©, it is easy to see 

P(V >v)&a 



(18) 



(19) 



where a =« m > 



is the normalizing constant. 
It is known (2j that, ki is typically large in practical power 
systems, and hence, N a could be approximated by a Gaussian 
random variable. Accordingly, the demand of opportunistic 
energy users D Q = N a E s follows a Gaussian distribution 
Af(q (v),a 2 (v)), with 



Plugging 



q a (v) = K 1 a v lo E 0: 
in (1161 . we obtain 



(20) 



2 We drop the suffix k for notational simplicity. 



R (ip ,s, u, v) = uD t - cis + c 2 Y + (v - c 2 ) q a (v) 

- (2TT)- 1/2 co- (v)exp(-y 2 /2) 

- c(Y-q (v))(l-Q(y)), 



(21) 
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where y = Y g . Denote the real-time pricing policy that 
solves the preceding equation by the mapping i? S;U : ijr — > v, 
where, recall that ip l is the system state {W,D t }. 

Using the statistical properties of the wind generation and 
the opportunistic user demand, reported in literature, we 
now further simplify R l (ip,s,u,v). According to lfl4l . the 
standard deviation of wind generation, is of the same order as 
the mean of wind generation. Typically, wind generation and 
the demand of opportunistic energy users are comparable. 
Observe from d20l l that the variance of the demand of 
opportunistic energy users is of the same order as its mean. 
We conclude that — has the same order as the square root 
of the average demand of opportunistic users. Thus, with 
Oy = of + a 2 , it follows that a\- ^> of. 

Recall that the opportunistic energy users are said to be 
relatively inelastic if — 1 < j a < 0, i.e., the percentage 
change in demand is greater than that of price; otherwise, 
they are relatively elastic. Since the price elasticity can have 
significant impact on a 2 and a\, we proceed to study real- 
time schedule for different cases of elasticity, with of- 3> of ■ 

Proposition 1. Suppose condition A holds. When the non- 
persistent opportunistic energy users are relatively inelastic, 
i.e., — 1 < 7 < 0, the real-time pricing policy is given by 

fls,u(4> 1 ) = Vcap- 

Proof: In practical power systems, according to |[T3ll , 
04], D t and W usually have continuous, symmetrical and 
unimodal probability distributions. Since a\ a 2 , intu- 
itively, there exists a finite constant Co > 0, such that: 



P {{Y - q (v) < -c a o (v)} 



U{Y - q (v) > c a o (v)}) « 1, 



(22) 



Q(-c )«l, Q(c o )«0, cxp(- C 2/2) w 0. (23) 

If Y — q (v) > cqo- {v), (f2Tb boils down to: 

R l (ip l ,s, u, v) = (v- (ci - c p )) q a {v) + (ci - c p )Y 

+ uDt-as; (24) 

When Y — q {v) < — caa (v), (fJTJ simplifies to 

R l {ip l ,s,u,v) = (v - c 2 ) q a (v) + uD t - as + c 2 Y. (25) 

It is clear that d24l i and d25l l are unimodal for v € 
[vmin,v C ap], both with peaks at v = v cap . This yields the 
real-time pricing policy: d SyU (^ 1 ) = v cap . ■ 
Remarks: Note that the result in the preceding proposition 
is intuitive, since, with the opportunistic users' energy de- 
mand being relatively insensitive (inelastic) to the announced 
real-time price, the scheduler can maximize profit by simply 
announcing the highest possible price, v cap . 

3 It is well-known that {23) is valid for cq > 3. 



Proposition 2. Suppose condition A holds. When the non- 
persistent opportunistic energy users are relatively elastic, 
i.e., ^jo < —1, the real-time pricing policy i? SjU is given by 



1+7° 

7o C2 



l+7o 



if Y> q ( 
if Y < q (j 



7q(ci 



l+7o 
7o£2\ 



Proof: When 7„ < —1, cr (v) can be expected to 
be much smaller than that in the inelastic case under the 
same real-time prices. With this observation, we resort to 
the certainty equivalence techniques [[TBI . By approximating 
D Q with its mean q (v), the profit in real-time scheduling is 
given by: 



R l (ij} l ,s,u, v) = uD t - as + c 2 Y + (v - c 2 ) q a (v) 
- c(Y-q (v))+, 



(26) 



which achieves the optimum at: 



7o(ci-c p ) 

l+7o 
7o£2 



7 (ei-c p ) ■ 



l+7o 

%HY) 



if Y > q ( 
if Y<q (^-) 
o.w., 



(27) 

which yields the real-time pricing policy -d StU in the elastic 
case. ■ 

Remarks: Note that the first case, i.e., Y > g ( 7 ° ( 1 c ^~ Cp) ), 
points to a case with energy surplus, i.e., there is more energy 
supply than the total demand from traditional and oppor- 
tunistic users, whereas the second case is tied to a case with 
energy deficit. Then, it is natural that the real-time price in the 
first case depends on the penalty c p in canceling a scheduled 
generation; and that the real-time price in the second case 
is a function of the cost of fast start-up generation. Note 
also that, in both cases, when the opportunistic users become 
increasingly elastic, i.e., j a — > —oo, the real-time price 
progressively decreases to the minimum allowable prices, i.e., 
ci — c p and c 2 , respectively. This monotonic behavior of the 
real-time price with respect to increasing elasticity comes at 
no surprise, since, as 7„ — > — oo, the average opportunistic 
user demand q (v) —> 0, i.e., the opportunistic energy users 
become more and more thrifty. Therefore, the scheduler must 
offer power at increasingly cheaper prices, up to the lowest 
possible price, to maximize profit. 

Having established an approximate real-time scheduling 
policy for both kind of opportunistic energy users, an ap- 
proximate day-ahead schedule can be obtained by solving 
the following single-stage optimization: 



V. 



b.U 



R l {i> l ,s,u,# s , u (>>p l ) 



(28) 



where R l is given by (O and the expectations depend on the 
exact stochastic models assumed for wind generation and 
traditional users' energy demand. 
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Proposition 3. The optimal decision of 



by 



non-pst « 



u<cap 

It 
l+7t 



Cl 



// - 1 < 7t < 

iy 7t < -i. 



S* = &rgmax{(c2-c 1 )S + KE'^ VD [(v-C2)D 
s 

- cl B (S - K(a t u*'> t + e t + D - W))}} . 

Proof: We first show that u depend on S and u only 
through s — Ejr> [Z? t ]. For convenience, define: 

s' = s-El t {D t ]. 

Since F = s' + W — it is clear from Proposition [TJ and 
Proposition |2 that the real-time pricing policy depends 
on the day-ahead decision only through s'. We denote this 
policy as ?? s / : (W, et) — > v. With this insight, by using the 
change of variable technique in ([Tol l, the objective function 
of TnJm— pst can ^ e rewritten as 



EwE 



R l {f,s',u,$ s ,{W,s t )) 



h(u) + h(s') + c 2 E w [W] 



where 



fi(u) = a t (u - ci)u 7t . 



(29) 



(30) 



f 2 (s')=E Wiet E 



■d,,(W,e t ) 
D 



[{ti s .{W,e t )-c 2 )D 



- cl B (s' -e t - D + W)]+{c 2 - ci)s'. (31) 

Let T denote the solution space for the objective function of 
^non-pst defined in |Z5). Thus 



{(u,s');u> 0,s' > -atu 11 }. 



(32) 



It can be verified that u* defined in the proposition statement 
maximizes f\{u). Define s' = — atu* 7t , and let s'* maxi- 
mize /2(s'). If we show that (u* , s'*) belongs to the solution 
space T, then (u* , s'* ) optimizes the day-ahead scheduling 
problem in (|29l . Since u* > 0, it is now sufficient to show 
that s'* > s' . A sufficient condition to establish this is given 
by 

Ms') < Ms' ), V s' < s' . (33) 

Under condition A, wind energy is not sufficient to meet the 
total energy demand when day-ahead price is u*, thus: 



W < (-s' + e t )+D o . 



Therefore, 



W + s' <e t + D 



V s' < s' . 



(34) 



(35) 



It follows that 1) W + s < D t + D a , i.e., there is no 
scheduled energy surplus, thus 1b = in d29i >: 2) Using the 
preceding statement, recalling the definition of Y, we see that 
Y < 1o( )■ Thus, from Proposition [2] the optimal real- 
time price $ s '(W,£t) turns out to be a constant jjp^-, i.e., 
independent of the system state and the day-ahead decisions, 
when the opportunistic users are relatively elastic. Also, for 



the relatively inelastic case, we know from Proposition[TJthat 
the optimal real-time price is a constant v cap . Letting i»o(7o) 
denote this constant real-time price for both the elastic and 
inelastic cases, respectively, we have 

h{s') = (C2 - ci)s' + (v (jo) - C 2 )qo(vo('Jo)), Vs'< s' . 

(36) 

Therefore, / 2 (s') < f 2 (s' ), Vs'< s' , and (u*,s'*) indeed 
lies in the feasible region T and hence optimizes the day- 
ahead scheduling problem in d29l . The optimal day-ahead 
decision, $*, can now be computed using s'* and u* . ■ 

Corollary 4. When the non-persistent opportunistic energy 
users are relatively inelastic, the complete two-timescale 
scheduling decision is given by: 

Ucap if - 1 < 7t < 
T+^Ci if 7 t < -1, 

S* = K (KiE v2Sp + atu* 11 - FJ, 1 (c p /c)) , 

where F^ 1 (-) denotes the inverse of the CDF of Z = W — e t . 

IV. Dispatch and Scheduling with Persistent 
Opportunistic Energy Users 

We now study multi-timescale dispatch and scheduling 
when the opportunistic users are persistent. Simply put, a 
persistent opportunistic user waits in the system for a new 
real-time price, in the next T2-slot, if the current real-time 
price is not acceptable. We assume that the opportunistic 
users are persistent across both T 2 and T\ slots, and that 
the opportunistic energy users that arrived in the day leave 
the system at the end of the day. 

Due to the persistent nature of the opportunistic users, 
scheduling decisions in both T 2 and T\ -slots affect the system 
trajectory and hence scheduling decisions in future time- 
slots, across both timescales. Thus, the scheduling prob- 
lem involves hierarchically structured control ifTBI . with the 
hierarchy defined across timescales. With this insight, we 
treat the scheduling problem as a multi-timescale Markov 
decision process (MMDP) IfTTl where decisions made in the 
higher level affects both the state transition dynamics and 
the decision process at the lower level, while decisions at 
the lower level affect only the decisions made at the upper 
level. The multi-timescale dispatch and scheduling problem 
at hand is particularly unique in the following sense: the two 
timescales do not overlap, since the upper level decisions 
(day-ahead) are made in non real-time. Thus the upper level 
scheduler does not have any direct observation of the effect 
it has on the lower level system dynamics, until the horizon, 
and make decisions solely based on stochastic understanding 
of the behavior of the lower level process. These properties 
make the two-timescale scheduling problem, with persistent 
users, uniquely challenging. We now describe the problem in 
detail. 



g 



For day-ahead scheduling, the scheduler decides the energy 
dispatch S m and the retail price u rn for the mth Ti-slot in the 
next day. Recall that the day-ahead scheduler has an accurate 
forecast of the expected amount of wind generation 9 m . We 
define the system state, ip m > corresponding to the mth Ti-slot 
in the next day as i/j'^ n = {^ TO ,P^}, where denotes the 
number of persistent opportunistic users carried over from the 
(m— l)th Ti-slot. During real-time scheduling, the scheduler, 
in each T2-slot, has knowledge of the wind generation and 
the demand from traditional energy users, along with the 
number of persistent opportunistic users carried over from 
previous T^-slot. Based on this information, it must decide 
a real-time price v k . m for the fcth T2-slot in the mth Ti- 
slot, i.e., the (fc,m)th T2-slot). We define the observable 
(observable in real time) state of the system in (fc, m)th T2- 
slot as ip l kim = {W k , m ,D tk , m ,P l k m }, where W k . rn denotes 
the wind generation in the fcth T2-slot in the ?nth Ti-slot , 
D tk is the energy demand from traditional energy users in 
the (fc, m)fh T2-slot, P k denotes the number of persistent 
opportunistic energy users carried over from the previous T2- 
slot to the (fc, m)th slot. Having explicitly defined the states 
of the system for the day-ahead scheduling and real-time 
scheduling, we introduce the optimality equations next. With 
X m = [Xi >m , X K ,m], we have 



V u (ib u 

v mlr r 



lax < 



E- 

W m ,D tn 



k.m 



-Er 



r" 



max {Ep 

v m m 



(38) 



'm+1 

As noted earlier, this is a MMDP over a finite horizon, with 
s m ,« m being the upper level (slower timescale) decisions 
and Vk,m being the lower level decisions. 

To mitigate the complexity of the MMDP problem, next 
we exploit the structural properties of the multi-scale dispatch 
and scheduling problem and recast it as a classic Markov 
decision process (MDP) (e.g., Ifl8l ). 

Proposition 5. With appropriately defined immediate reward 



-RJJj and action space a^, the two-level scheduling problem 
can be written as a classic MDP at the slower time-scale, as 
below: 

= max teftO 

We now proceed to discuss the transformation of the 
two-level scheduling problem from a MMDP to a classic 
MDP. In the two-level scheduling problem, recall that the 
lower level decisions are essentially the mapping from the 
realizations of wind energy, traditional users' energy demand 
and persistent opportunistic users to the real-time price, 



i.e., Cfc 



{Wk, m , D tk P l k m ] -> v k , m . Consider a 



stationary real-time pricing rule within each Ti slot, i.e. 
Ci,m = (2.m ■ ■ ■ (K.m, and denote this stationary mapping 
by ( m . A key step is to view £ m as an action at the day- 
ahead scheduling level, in addition to actions s m ,u m . With 
this insight, we can simplify the MMDP into a classic MDP. 
We elaborate further on this below. 

The expected net reward from slot m € {1, . . . , D} until 
slot M in the day-ahead market is thus given by the Bellman 
equation: 



max 

-{s m ,u m ,C, 



Cfi(C+i)} 



(39) 



where, recall, E\ refers to the expectation over x conditioned 
on y. The expectations used in the MDP formulation are 
explicitly provided in d3Tb . The terminal reward, V^, is given 
by 



™*RM^M,a u M ) 



(40) 



Note that the immediate reward corresponding to mth Ti-slot 
is a function of the realized values of wind generation mean 
(that is accurately forcast 24 hours ahead) and the number of 
persistent opportunistic users carried over from previous Ti- 
slot. We now proceed to explicitly characterize the immediate 
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(•) 

E p k r (.) 



1+1 

-1 1 L- r 



{W kim ,D t .,N Km ,Pl m } 



(•) 



n 



i>k,i 



E 



(•) 



K Pl „ ^Pi 



pi pi 

,Elf-^E P p l-L) 



Ew kim E Dt E Nk __E 



K,m - 

{W k<m ,D t .N Km ,P l h m } 



(•) 



N k . m +P l k m f 

E ' ( S fc,m (i-\J^(n,,J 

P(V < v k , m ) 
aWk,m,D tk>m ,P l ki J 
N k , m +Pl m , . 



(JVfe, 



E 



No. 



Na k 



,-Pl 



»(•) 



(n 8 ,. w )^.»(i-n Bfcim ) ( ^ + ^--^.- ) (.) (37) 
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reward R^ n , for me {1, . . . M}: 

= Vl m ({9 m ,P^ m },a u m )\k=i (41) 
where, P\ m = P"i by definition, and for k G {1, . . . , K}, 

^fe,m({^mi ^fc.mll a m { s mi u rai Cm}) 



V. Numerical Results 

For concreteness, we now study, via numerical exper- 
iments, the performance of the proposed multi-timescale 
dispatch and scheduling policy. First, we define profit margin 
as the ratio of the average net profit to the sales in a T 2 -slot. 
We assume the opportunistic users are non-persistent. 



E 

+E 



w k , ... 
Pi, 

>! 



{W k m , D t , Pl m }, dm 



^+l.m({^mi Pk+l,m}: a m); 



(42) 



where V/o„ is given by 

Vk, m ( { ®m , P\c, m } ) a m. 



i? 



if.mi u, m) 



(43) 



Note that the quantities R l k m and m can be regarded as 
the immediate reward and the net reward at the lower level 
MDP, if the problem is viewed as an MMDP. The quantity 
R l k m is a function of the realizations of the wind generation, 
the demand from traditional energy users and the number 
of persistent opportunistic energy users carried over from 
previous T2-SI0L More specifically, we have that 



^fc,m('/ , t,mi a m) 

= u mDt kt „ + Ejy km E Na ^ 

+1a(— CpS m ) + 1b(— ek jm Cp 



(8, 



Vk,i 



m)Cl) 



(44) 



where Nk, m denotes the number of opportunistic users ar- 
riving at the (k, m)th slot, and N ak m denotes the number of 
opportunistic users that become active in slot (k,m). D 0k m 
denotes the energy consumption by the opportunistic users 
in slot (k, m). Recall that E a denotes the energy demand per 
active opportunistic energy user. Thus, we have 



D„ 



N a ,_E n 



(45) 



The quantity ek, m denotes the surplus energy, given by 

£fc,m = Wh,m + 

is given by the mapping (. 



-{D t ,+D ,J. (46) 



Recall, the real-time price v k 

as 



Vk,; 



tm{{W k<m ,D tk 



> -ffe,m})- 



(47) 



The indicator functions 1a, 1b and lc correspond to the 
various deficit/surplus energy states, and were explicitly 
defined in Section III. 

Summarizing, we have shown that the two-timescale dis- 
patch and scheduling problem with persistent users can be 
recast as an MDP with continuous state and action spaces. 
Using appropriate discretization techniques, we can reformu- 
late it as a classic discrete state and action space MDP, which 
can be solved optimally or near-optimally using various 
solution techniques available in the literature |fl9l . 
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Fig. 3. Profit margin at various penetration levels of wind generation and 
opportunistic demand (70 = —2) 

In Fig. [3] the profit margin is plotted against various values 
of the penetration of opportunistic users and penetration of 
wind energy in the system with two-timescale scheduling. 
We use price elasticity of —0.5 for traditional users and 
—2 for opportunistic users. As expected, the penetration 
of wind energy increases the profit margin since wind en- 
ergy is harvested 'cost-free'. In contrast, as the demand 
of penetration of opportunistic energy users increases, the 
profit margin decreases. To explain this, we first identify 
the two principal events that lead to losses in the system 
operator's profit: (a) The penalty (c p ) when a fraction of 
the dispatched conventional generation is reverted; (b) the 
loss of revenue when the dispatched energy S is insufficient 
to meet the total energy needs and the operator purchases 
energy from fast-start up generators at higher cost c 2 . The 
uncertainty in opportunistic energy users' demand is one of 
the factors that could lead to either of these events. An 
over-estimation of this demand in the day-ahead schedule 
leads to event (a), while an underestimation of this demand 
leads to event (b). In our experiments, since the demand 
from traditional users is deterministic, the higher the demand 
from opportunistic users, the higher the uncertainties on 
the demand side and hence the higher the losses at the 
system operator. This insight further underscores the need 
for efficient pricing mechanisms that intelligently tackles the 
ever-increasing uncertainties in the power system. 

Fig. [4] compares the profit margins with and without 
multi-timescale pricing for various values of wind energy 
penetration levels, i.e., the ratio of the wind energy generation 
to the total energy. In the benchmark system, without multi- 
timescale price, all the users exhibit traditional response 
to prices with the same price elasticity 7. The prices are 
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Fig. 4. Profit margin with various elasticity (r u = 20%) 

optimized by taking into consideration the statistics of the 
traditional users' demand and wind generation. Considering 
the multi-timescale scheduling system, for fair comparison, 
we assume that the opportunistic users have the same elas- 
ticity as the traditional users in the system. We compare the 
two systems for different values of 7. As expected, for a 
fixed wind penetration level, for the same value of 7, the 
profit margin is higher when real-time pricing is employed 
to mitigate the uncertainties in demand and supply. Also, note 
that, as the price elasticity increases, the opportunistic users 
exhibit increasingly thrifty behavior essentially reducing the 
profit margin for the operator. In addition, as expected, the 
profit margin increases with wind penetration with or without 
real-time pricing since wind energy is assumed to be cost- 
free. 

To summarize, numerical results suggest that two-time 
scale scheduling effectively addresses the volatility of en- 
ergy generation and the uncertainties in the demand from 
opportunistic users. Additional insights include the following: 
the profit margin of system operators increases with the 
penetration level of wind and decreases with demand from 
opportunistic users and that the elasticity of the opportunistic 
users plays a major role in the power system design. 

VI. Conclusion 

Wind generation is among the renewable resources that 
has most variability and uncertainty, and exhibits multi- 
level dynamics across time. Aiming to tackle the challenge 
of integrating volatile wind generation into the bulk power 
grid, we study multiple timescale dispatch and scheduling, 
for a smart grid model, via day-ahead scheduling and real- 
time scheduling. In day-ahead scheduling, with the statistical 
information on wind generation and energy demands, we 
characterize the optimal procurement of the energy supply 
and the day-ahead retail price for the traditional energy users; 
in real-time scheduling, with the realization of wind gener- 
ation and the load of traditional energy users, we optimize 
real-time prices to manage the opportunistic energy users so 
as to achieve system-wide reliability. More specifically, when 



the opportunistic users are non-persistent, we obtain closed- 
form solutions to the multi-scale scheduling problem. For the 
persistent case, we treat the scheduling problem as a multi- 
timescale Markov decision process, and then we show that it 
can be recast, explicitly, as a classic Markov decision process. 

We believe that the studies we initiated here on multi- 
timescale dispatch and scheduling for integrating volatile 
renewal energy into smart grids, scratch only the tip of 
the iceberg. There are still many questions remaining open 
to improve the penetration of renewable energy into power 
grids, and we are currently investigating these issues along 
this avenue. 

References 

[1] J. Chow, W. De Mello, and K. Cheung, "Electricity market design: An 

integrated approach to reliability assurance," Proceedings of the IEEE, 

vol. 93, pp. 1956 -1969, nov. 2005. 
[2] S. Newman, "Smart appliances for an energy-efficient future." 

http://green.yahoo.com 2008. 
[3] R. Sioshansi, "Evaluating the impacts of real-time pricing on the cost 

and value of wind generation," Power Systems, IEEE Transactions on, 

vol. 25, pp. 741 -748, may 2010. 
[4] M. Albadi and E. El-Saadany, "Demand response in electricity markets: 

An overview," in Power Engineering Society General Meeting, 2007. 

IEEE, pp. 1 -5, 24-28 2007. 
[5] J. Bialek, P. Varaiya, and F. Wu, "Risk limiting dispatching smart grid." 

Talk at CNLS Smart Grid Seminar Series, 2009. 
[6] F. Bouffard and F. Galiana, "Stochastic security for operations plan- 
ning with significant wind power generation," Power Systems, IEEE 

Transactions on, vol. 23, pp. 306 -316, may 2008. 
[7] L. Xie and M. Ilic, "Model predictive dispatch in electric energy 

systems with intermittent resources," in Systems, Man and Cybernetics, 

2008. SMC 2008. IEEE International Conference on, pp. 42 -47, 12-15 

2008. 

[8] A. J. Wood and B. F. Wollenberg., Power generation, operation, and 

control. New York : J. Wiley Sons, 1996. 
[9] J. Hetzer, D. Yu, and K. Bhattarai, "An economic dispatch model 
incorporating wind power," Energy Conversion, IEEE Transactions on, 
vol. 23, pp. 603 -611, june 2008^ 

[10] Y. L. Xu and J. Chen, "Characterizing nonstationary wind speed using 
empirical mode decomposition," Journal of Structured Engineering, 
vol. 130, no. 6, pp. 912-920, 2004. 

[11] H. Allcott, "Real time pricing and electricity markets." Harvard Uni- 
versity, Jan. 2009. 

[12] R. Boisvert, P. Cappers, B. Neenan, and B. Scott, "Industrial and 
commercial customer response to real time electricity prices." available 
online at http://eetd.lbl.gov/ea/EMS/drlm-pubs.html, Dec. 2004. 

[13] S.-E. Fleten and E. Pettersen, "Constructing bidding curves for a price- 
taking retailer in the norwegian electricity market," Power Systems, 
IEEE Transactions on, vol. 20, pp. 701 - 708, may 2005. 

[14] H. Agabus and H. Tammoja, "Estimation of wind power production 
through short-term forecast," Oil shale, vol. 26, no. 3S, pp. 208-219, 
2009. 

[15] H. Van de Water and J. Willems, "The certainty equivalence property 
in stochastic control theory," Automatic Control, IEEE Transactions 
on, vol. 26, pp. 1080 - 1087, oct 1981. 

[16] M. Mahmoud, "Multilevel systems control and applications," IEEE 
Trans. Syst, Man, Cybern., vol. SMC-7, jun 1977. 

[17] H. S. Chang, P. J. Fard, S. I. Marcus, and M. Shayman, "Multitime 
scale markov decision processes," Automatic Control, IEEE Transac- 
tions on, vol. 48, jun 2003. 

[18] M. L. Puterman, Markov decision processes: discrete stochastic dy- 
namic programming. New York : J. Wiley Sons, 1994. 

[19] E. A. Feinberg and A. Shwartz, Handbook of Markov Decision 
Processes. Springer Netherlands: International Series in Operations 
Research and Management Science Series, jun 2005. 



